03 Alignment and Preprocessing ¶

Once the data is made available as detailed in Data Ingestion , the next step is to ensure the data has been appropriately reshaped and aligned across data sources for consumption by the machine learning pipeline (or other analysis or simulation steps).

We'll be aggregating data across several years, using the same Landsat images used in the Walker Lake example. See that topic for more work on calculating the difference between the water levels over time.

Recap: Loading data ¶

We can use this EPSG value shown above under the crs key to create a Cartopy coordinate reference system that we will be using later on in this notebook:

Preprocessing ¶

The first step in processing data is to remove the missing values. In this case xarray self-reports the values assigned to nodatavals . We can use this information to set the missing values to NaN .

We can make sure that no more -9999s show up in the data, by calculating the minimum value in each DataArray as follows:

NOTE: These operations take a non-trivial amount of time because they require that the data actually be loaded.

Compute NDVI ¶

Now we will calculate the NDVI (vegetation index) for each of these image sets.

NOTE: To calculate NDVI we have selected based on the band label. In xarray you can also select by index.

Combine data from overlapping grids ¶

These two sets of Landsat bands cover roughly the same area but were taken in 1988 and 2017. While they have the same resolution (30m) and the same grid, they have different numbers of grid cells per image and different x and y offsets. We can see that by printing the first x value for each year and seeing that they are not equivalent.

Treating year as DataArray name ¶

XArray allows arbitrary data to be combined into one xr.Dataset , though if the underlying grids are not aligned, then the shape of the dataset grows to be the union of the dimensions on each array. Here we'll combine the data from different years, providing the year as the name of the underlying DataArray:

We can quickly sample one point, such as the center of Walker lake: latitude 38.6942° N, longitude 118.7081° W. First convert the lat, lon (PlateCarree) values to the coordinate reference system of the data (using the crs var we defined above):

Then we'll select the data point nearest to this point, and use .compute() to actually load that data using dask :

Select region of interest ¶

We'll use the area around the central point as the Region of Interest (ROI). In this case we'll use a 30 km box around the center point.

Since the data are on the same coordinate system, when these data are visualized, the plots can be linked. For instance, if you pan one of the plots, the other will pan as well:

We can also subtract the two ROIs, to visualize the difference between 2017 and 1988:

Treating year as coords ¶

Another way to join the data from the two different years, is by treating the years as coordinates. This approach is more logically sound, but sometimes global attribute data can be lost. For instance if the crs were different on the two dataarrays then that attribute would not appear on the output. To work around this, it is sometimes helpful to assign any attrs that differ to coords . This would look like NDVI_2017.assign_coords(crs=NDVI_2017.attrs['crs']) .

We'll do the same sampling at a point and select a ROI to demonstrate how years as coords differ from years as names.

NOTE: It is simpler to define a series of subplots where the variable that is being iterated over is a coordinate.

Since the output is a gridspace, we can select a a subplot from the output and alter it in place. For instance, we only really need the colorbar on the second subplot, so we can turn it off for the first one:

Now we can compute the difference:

Aligning data on different grids ¶

Suppose that the two sets of Landsat images are on a different grid, which often happens because of differences in how data is collected and processed. To illustrate how to handle this case, we'll manually shift the 2017 coordinates slightly:

It is still possible to concatenate the arrays, but the x and y coordinates are now 2 times greater than when the arrays are on the same grid.

These data can be plotted just as before:

Since the years don't share coordinates, the difference is all NaNs

To resolve this we can declare a new grid that spans the entire ROI with the same resolution as the original arrays (30m)

We will use linear interpolation to calculate the values for each grid cell. This operation is not yet supported for dask arrays, so we'll first load in the data.

Now we can use the new coordinates defined above to interpolate from the input coordinates to the new grid. The options are nearest and linear with linear being selected by default.

And now we can plot the difference between the two years:

Next: ¶

Now that the data have been aligned and preprocessed appropriately, we can then use them in subsequent steps in a workflow. For instance you might need to learn more about Resampling . Or you might already be ready for Machine Learning .